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ABSTRACT 

This paper presents the findings of a qual i tative 
followup survey of a sample of programs listed in the 1986 "Directory 
of Evaluation Training Programs." Four programs listed in the 
directory are no longer considered evaluation training, and five, 
originally similar, are still going strong. A sampling strategy was 
used tc compare the discontinued programs with the continuing 
programs through discussions with program administrators. Comparison 
indicated that several areas are important in program continuation. 
To begin with, two programs no longer listed were actually in 
operation with only minimal change, a finding that suggests that the 
present directory may underestimate the number of viable programs. 
The perceived importance of the program is vital to its continuation. 
Faculty members and their diverse interests are keys to continuing a 
program. Grant funding can be a help in getting started, but it can 
be risky if it does not eventually result in program 
institutionalization and ongoing support. Student recruitment is a 
necessity for program continuation. Overall, existing evaluation 
training programs seem to be meeting the challenges of training more 
and better e/aluators. An appendix describes the nine programs. 
(Contains 18 references .) (Author /SLD) 



ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft Vc ft ft ft ft ft ft ft 

* Reproductions supplied by EDRS are the best that can be made x 

* from the original document . ,c 

ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft * A ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft ft Vc ft ft ft ft ft ft 



4 



CO 

Q 



A qualitative follow-up of the survey for 
the 1994 Directory of Evaluation Training 

Programs* 

for the Annual Meeting of the 
American Evaluation Association 
Boston, Massachusetts 



U S DEPARTMENT OF EDUCATION 
Othce o* Educational Rewerch and Improvement 
EDUCATIONAL RESOURCES INFORMATION 

/ CENTER (ERIC) 

p/ovs document has been reproduced as 

reTe.vecnrom me person or ornam/afon 

originating it 
C Minor changes have Oeen made to improve 

reproduction Qu»l«W 



• Po-nts ol v,ew or opm.ons stated -n ih, s ;ooc u 
mem do not necessity represent oliicai 
OERI positron or pot'Cy 



by 

Crystine Durrant 

Graduate Student 
and 

Nick Eastmond 

Professor 



PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER .ERIC. 



Department of Instructional Technology 
Utah State University 
Logan, Utah 84322-2830 

Tel. (801) 797-2642 
FAX (801) 797-2693 
E-mail: NEAST@CC.USU.EDU 



November 5, 1994 



1 2 

erJc best copy available 



Abstract 

This paper presents the findings of a qualitative follow-up survey of a 
sample of programs listed in the 1986 directory, four of which are no longer 
considered evaluation training, and five others being originally similar and yet 
going strong. 

A description of the procedure used is presented, with its strengths and 
weaknesses. The findings are presented with the programs surveyed being 
anonymous. Recommendations are made, such as retain faculty, seek stable 
sources of funding, take a proactive stance, and recruit students. Potential pitfalls 
for programs are identified, being decreased support from administration, faculty 
not being replaced, grants ending, and faculty overload. 



Introduction 

Evaluation means many things to many people. To some it means giving 
standardized tests; to others, the image of quality control comes to mind. One of 
the most popular definitions comes from Worthen and Sander's (1987) book. It 
states that evaluation is simply the method of determining worth (p 22). In 
determining worth, much depends upon the credibility of the evaluator. He or she 
is stating whether the object or idea is any good or not. The results will only be 
accepted it the evaluator is perceived to be competent. The evaluator needs to be 
taught to do a quality job that will speak well of both the profession and the 
professi' nal. 

Many evaluation training programs exist to train professionals in various 
aspects of evaluation. Some consist of a single course in a sequence of study 
(Morris, 1994; Wortman and Yeaton, 1986), while others are entire areas of focus 
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(May, Fleischer, Scheirer, and Cox, 1986). In 1986, the Evaluation Research Society 
and the Evaluation Network produced a Directory of Evaluation Training 
Programs. It reported 46 different programs. In an effort to prepare the updated 
1994 American Evaluation Association (AEA) Directory of Evaluation Training 
Programs, Dr. James Altschuld, together with Engle, Cullen, Kim, and Macce, 
surveyed 77 possible evaluation training programs identified in the United States. 

This survey defined an evaluation training program as consisting of 
"multiple courses, seminars, practices, offerings, etc., designed to teach primarily ... 
evaluation principles and concepts (Altschuld, etc., 1993, p 1)." It covered such 
areas as background information, characteristics of students or trainees, program 
description, and other institutional offerings. When Altschuld and Engle reported 
on their findings at the 1993 Annual AEA conference, it became apparent that ten 
of the original 46 programs in the 1986 directory would not appear in the new 
directory. This was an area of concern that we have undertaken to investigate. 

Evaluation training programs are likely to cover much the same material, 
such as the history and philosophy of evaluation, different approaches to 
evaluation, and techniques of evaluation. What is more important, the real 
accomplishment of evaluation courses is to help the trainees develop evaluative 
sensibility, by using their values and judgments (Davis, 1986; Mertens, 1994; 
Sanders, 1986). One of the best ways to do so has been to allow students to obtain 
hands-on experience as they engage in coursework (Eastmond, Saunders, and 
Merrell, 1989). Evaluation training courses help train good evaluators. Usually 
when a single course is used, the intent is not to train full-fledged evaluators, but 
rather, as Morris (1994) notes, to train educated consumers of evaluation or 
cooperative advocates who can work alongside a practicing evaluator. The other 
option of pure apprenticeship work with an evaluator is not practical for many 
people. 
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Problem Statement 

Ten of the original 46 programs in the 1986 Directory published by May, Fleischer, 
Scheirer, and Cox no longer qualified for inclusion in the directory by the time 
Altschuld and Engle set about publishing a second directory based on the 1993 
survey. Because that survey could only obtain limited information regarding each 
program, many specific questions remain unanswered. Some of the proposed 
reasons for program demise included a program being run by a single person, and 
when that person retired, the program ended; faculty or administrators sabotaging 
the program; budget cuts that the legislature enforced on the school; or perhaps the 
program had changed its name and not have been reached by the surveyors. 

Specifically, the professional association of evaluators (AEA) lacked 
empirical data to explain why some programs declined and others flourished in 
this seven year period. 

Purpose and Objective 

The purpose of this project was to identify pertinent information about 
evaluation training programs that have ended recently. The Altschuld & Engle 
study involved a broad survey of programs. In this study we wanted to investigate 
programs in more depth, to understand more about the human dimension of 
specific programs. We compared four such discontinued programs to five 
continuing programs that started roughly at the same time as their counterparts. In 
particular, we were interested in how programs began, how they developed and if 
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there was a "critical mass" of faculty and students needed for a program to "take 
off." This was accomplished with the use of a telephone survey. 

Review of the literature 

Formal evaluation training programs really began some almost 30 years ago 
when federal educational grants began to require some type of evaluation and 
accountability (Altschuld and Thomas, 1991). During the past thirty years, 
evaluation has become, if not a professional field, then at least a professional 
emphasis in many fields (Eastmond, 1992; Merwin and Wiener, 1985; Worthen, 
1994; Worthen and Sanders, 1984). Its training has gone from a hands-on, trial-and- 
error experience, to single courses, to an expanded curriculum at many schools of 
higher learning (Altschuld and Thomas, 1991). Unusually, perhaps, there is some 
disagreement as to which discipline it should be taught in. 

Worthen and Sanders (1987) argue that "evaluation ... necessarily cuts across 
disciplines, and evaluators are thus denied the luxury of remaining within any 
single inquiry paradigm (p. 53)." As such, in the education process, 
it is best that several disciplines be sampled. Only through an 
interdisciplinary education can the evaluator become sensitive to the 
wide range of phenomena to which she must attend to if she is to 
properly [conduct evaluation] (Worthen & Sanders, 1987, p 33). 
Eastmond also feels that it is "impossible to create" the expertise needed "within a 
single discipline." The "ideal program would be cross-disciplinary (Eastmond, 
1992, p 5)." Likewise some feel that evaluation should be integrated as much as 
possible into core courses taught in any program (Perloff and Rich, 1986), while 
others feel that evaluation should be taught in an entirely separate department. 
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(For example, see the programs described at the University of California, Los 
Angeles, in the Directory published by May, Fleischer, Scheirer, and Cox, 1986.) 

Moreover, those who are not professional evaluators are also taught 
evaluation. Managers are taught evaluation in the process of budgeting (Perloff 
and Rich, 1986), while health professionals learn evaluation for health 
administration and health services research, as well as in vocational education 
(Eldredge, Fried, and Grissom, 1991; Wortman and Yeaton,1986). Educators learn 
evaluation through formal and informal experiences (Sanders, 1986), and 
engineers study evaluation as a part of quality control and design analysis. 
Evaluation is an integral part of virtually all professions. 

Models for teaching evaluation have come from education or psychology 
(May, Fleischer, Scheirer, & Cox, 1986), as well as sociology or the legal education 
method (Caulley and Dowdy, 1986). No one field has a monopoly on evaluation or 
the teaching of it. 

Finally, evaluation is not something you can learn once and never again 
open a book to study. It is, or should be, a "lifelong pursuit (Eastmond, 1992, p 4)." 
As new methods appear there is, at times, almost a "dire need for additional 
training (Eldredge, Fried, and Grissom, 1991, p 11)." Professional organizations and 
journals can be an integral part of this training (Worthen and Sanders, 1984). 

In short, evaluation is a general practice that mainly concerns a frame of 
reference. Saying "I am an evaluator." may be equivalent to saying "I am a worthy 
judge." This self-qualification will generally be more acceptable to its recipients if 
backed up by training through a program or organization. Evaluation is a field of 
endeavor that should be treated with respect and mastered well with the proper 
training. 
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Procedure and Methodology 

Method: 

A purposive sampling strategy was used. The previous researchers were 
asked to identify selected programs where the contrasts between those ending and 
those continuing would be most evident. Four programs that were no longer 
included in the directory (discontinued programs) were selected, and five other 
programs that were still ongoing (continuing programs) were selected for 
comparison. We mailed out the questions we would be asking to the contact 
persons, and followed up with the actual telephone call roughly one week later. 
We took notes during the conversation, (no audio recordings were made), and 
asked for other sources of information and other people to contact. We wrote up 
the notes about all the conversations within 24 hours, and afterwards sent them to 
the contact person for confirmation of their accuracy when all the contacts have 
been made. This last step in still in process, with only one contact having sent back 
the corrected description. 



Procedure: 

The initial step was to become familiar with the literature, secure the 
funding, end clarify the aims of the project. It was during this time that the 
questions and programs were identified. 

The main work involved making telephone calls to the nine programs and t 
writing up the results. The following questions were sent to the contact persons in 
letter. They were then contacted by telephone and asked to respond in person. 
Respondents were assured of the anonymity of their responses, both for individual 
program identification. 

1. Describe your involvement with the program. 
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2. What were the major steps in establishing it? Developing it? 

3. What arc some of the factors that led to the program being whe^e it is today? 

4. Are there other people who would provide a different perspective or additional depth? 

5. Do you have a pamphlet that could help us understand the official position of your 
program? 

6. Do you have any additional concerns or information to add at this time? 

The responses were noted by hand and then complied into a summary for each 
program. Each program received at least one phone call, with an average of three 
telephone calls, and two letters, one prior to the interview stating we would be 
calling, and the other after the interview, asking for an accuracy check. We asked 
the contact person or persons for names and phone numbers of other people who 
might be knowledgeable about the program or provide a different perspective on its 
history. We then called the new references. They in turn were asked for other 
people to call. When the information was repeated or we talked key people, 
(generally 2-3 per program), we wrote up the conversations. 

The last, wrap-up stage was the time to clarify the findings and put them in a 
cohesive form. It is hoped that the results are applicable to at least 36 evaluation 
training programs. 

Results 

During the period from March 9 to October 25, 1994, we were able to contact 
representatives of all nine programs. People were fairly eager to talk about their 
programs, especially the students. In some cases, trying to match up schedules 
proved too frustrating to the respondent, and he or she simply mailed back their 
annotations on the questionnaire that had been sent to them. In one case, the 
correspondence was carried on through e-mail. Every program, with one 
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exception, received at least three telephone interviews. Some staff members were 
eager to talk about their program, and were well informed. In a few cases, 
respondents were ill-informed or refused to comment. In these cases, efforts were 
made to locate another respondent. The method was time consuming, but less 
expensive than some other forms of interviewing. The telephone had an 
advantage in that people didn't mind talking, as opposed to writing, and that the 
interviewer was able to probe on topics that might be otherwise have been 
misunderstood. 

Altogether we talked to 20 faculty, 4 administrators, 4 students, and 6 staff 
members. After talking with all 34 people, the following conclusions about 
programs can be drawn: 

Discontinuing programs: Two programs that had been assumed to be 
discontinued were still functioning, one having been overlooked in the previous 
study and the second excluded because of the change in classifying criteria. In this 
program a single course in evaluation continues to be taught. The current 
definition requires multiple courses for inclusion as a "program." One of the two 
remaining discontinued programs had been depending upon grant funding which 
expired, after which two faculty members retired, one died, and the final person 
moved to another school so that although the department is still there, the 
emphasis is not. The final school, previously considered leading program in the 
field, had a program, but in one year it saw two faculty move away, one move to 
another department, and six years later, the remaining faculty member moved to 
another institution. After he left, the program was defunct. 

Continuing programs: All five started at about the same time (late 
1970's, early 1980's) "when evaluation was coming of age." In every case, the 
programs have evolved to include more course and program offerings, more 
depth, and more field experiences. Courses have evolved in the direction of more 
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hands-on experience, more paid internships, and more contact with professional 
organizations like AEA. In no case was a continuing program found to have been 
misclassified. 

Discussion 

The discontinued programs had an average of two contacts each. They 
revealed several areas for thought, including sampling problems, perceived 
importance of the program, the importance of faculty, and the need for a stable 
source of funding. These problems' are expanded below. ' 

Sampling problems: The classification system for the study is not 
without its faults. Two programs identified as having been discontinued were in 
fact continuing on with minimal change (one seemed to have been improved). 
Since there were no instances of a misclassification of continuing programs, it is 
likely that the present directory tends to underestimate the number of viable 
programs. 

Perceived Importance: If a program is seen as peripheral to the 
institution, department, or school, it is probably vulnerable, with budget and 
personnel cuts shortly to follow. 

Faculty: Faculty are key to beginning and ending a program. "When 
you're out of faculty, you're out of business." 

Soft money: Grant funding can be a help in getting a program started, but 
it can also be risky if it does not eventually lead to program institutionalization, 
with support coming from an ongoing source. 
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The continuing programs were also interesting. They each had a minimum 
of three contacts and some form of brochure that explained their programs in terms 
of required courses, time, and opportunities. Some of the areas that came to mind 
as being worthy of calling attention to included diversity in faculty interests, a 
proactive stance, stable funding sources, faculty retention, faculty overload, and the 
recruitment of students. These ideas are expanded below. 

Diversity of faculty research interests can make for a strong 
program, if faculty interests complement each other and if there is mutual respect 
for varying points of view. One faculty member stated: "Students struggle to 
understand 'who's right.'" As they do that, they have to wrestle with different 
points of view. 

The faculty interests are really what shape the program. One program was 
originally an educational research program, but the faculty were interested in doing 
and teaching evaluation, and eventually the whole program was transformed into 
an evaluation training program. 

Moreover, the prominence of the program comes from the faculty. Faculty 
are encouraged to perform evaluations and do research , so that what they teach 
has a practical side. Most students and faculty believed that the quantity and quality 
of faculty publication were a source of strength in their program. The 
administrators only said that the faculty are the strength of the program and did 
not elaborate. 

Proactive Stance: A program needs to examine itself for strengths and 
weaknesses and then work to strengthen the program before some administrator 
takes action. A program should have its own agenda for development, possibly 
formalized as a strategic plan. One program saw itself as weak in one area and 
deliberately hired a new faculty member to fill that need and help them to stay on 
the forefront of technology and new ideas and perspectives. 
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Funding sources: Outside sources are important in funding students. 
They allow intern options and full-time student status, also allowing classes to be 
tanght during the day, rather than in the evening, Several of the programs 
contacted had their own center for contracting projects; others had outside grants 
and interested businesses. 

Recognizing that many students do have jobs and encouraging them to 
attend class at night, though, can greatly increase program enrollment, It also 
allows professionals to continue their work but at the same time to pursue their 
education. 

Some programs have had to justify their existence; some will have to in the 
future. One faculty member stated that he saw the coming need defend 
expenditures with the state. Another faculty member was worried that, although 
their program had nearly a quarter of the students in the department, they weren't 
getting the proportional amount of funding. 

Faculty retention: In insuring continuation, early retirement can be an 
attractive option for faculty and can play havoc with program continuation. It can 
be difficult to insure that, as a faculty member, you are replaced as you leave. 
Emeritus status helps in advising students, but not in continuing a program. 

Faculty overload: Outside sources of work and funding can be so 
demanding that a professor no longer has time to teach, and is reduced to an 
advising position. This was the case for at least one faculty member talked to. 

Regarding recruitment of students: It appeared that students 
learned of programs through diverse means, and then decided that they were 
interested (or well suited by temperament) in studying in the area. Quality of 
faculty (leading edge research interests and skills) was seen by the faculty and 
administrators as important to attracting students. Students, on the other hand, all 
said they "fell into" their programs, discovered them by accident, and then decided 
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they wanted to peruse the program. At least one faculty member mentioned that 
his program had no recruitment whatsoever. One solution offered was that the 
graduates were going out into diversified areas: therefore, the program's name was 
being spread across the country and into other countries as well. 



Conclusion 

Both students and administrators occasionally had a hard time seeing the 
program. One faculty member said this was because his program was not showy 
like other programs in his department. Active recruitment may be a way of 
increasing visibility for a program. 

On the other hand, all of the students and most of the faculty mentioned 
that their individual program was good because the faculty, and occasionally the 
students, published papers and spoke at conferences. This professional exposure 
was seen as a major draw to attracting students, and may be a way to attract support, 
both moral and financial. 

A concern voiced by several faculty had to do with retirement. They were 
concerned that when they did retire, their positions would not be filled and the 
program would diminish. Careful recruitment and continuing effort to save a 
"faculty line" is critical to preserving programs. 

The use of grant funding to establish and Maintain programs can be seen as a 
mixed blessing. It may help establish a "presence" for a program and may help 
provide opportunities fo: students. However, if a degree of intsitutionalization 
and local commitment does not emerge over time, the program will likely vanish 
when the outside funding diminishes. 
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All in all, the existing evaluation training programs seem to be rising to the 
task of training more and better evaluators. The future looks good. 
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Appendix A 

Descriptions of the nine programs surveyed 
a - Discontinued programs 
b - Continuing programs 
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Discontinued programs: 

Having started in the mid 70's,the evaluation program at Book 
University was considered a leader in the field in the early 80's. It offered 
such classes as evaluation research, statistics in experimental design, and 
seminar on evaluation research. A field practicum formed a major part of 
the program, along with collaborative projects between faculty and students. 
It flourished until three of the five faculty either went to other universities or 
to migrated to other departments. Since evaluation was then viewed as 
peripheral to the progress of the department, the faculty were not replaced. It 
limped along with only one faculty member until he left for another 
institution in 1989. It is now considered defunct. 

The evaluation program at Flame Graduate School is still in good 
order. It was apparently overlooked (or the inquiry lost in the mail) in the 
survey revising the AEA directory. The program started in the late 1970's 
with a seminar on evaluation and has grown to include numerous course 
offerings, field placement supervision, meta-analysis, and advanced 
methodological issues. The program reports approximately 30 doctoral 
students taking courses. 

The University of Three Peaks no longer offers an evaluation degree. The 
program was completed on five-year grant schedule in the 1980's. The program is 
only a dim memory now. The Student Services Coordinator at UTP insisted that 
the program had ended before 1985, leaving no memorabilia. The faculty have all 
retired, moved away, or passed on. 
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Although Wolf College was listed in the 1986 Directory of Evaluation 
Training Programs, it is not listed in the new 1994 Directory. The College is 
still there and the course offerings are still there. The instructor of the 
evaluation course, our main contact, has written a well-known book on 
evaluation. The reason the program is no longer listed is that the definition 
of evaluation training programs has changed to "multiple courses, seminars, 
practicums, offerings, etc. designed to teach primarily ... evaluation principles 
and concepts." (Altschuld, etc., 1993) Wolf College's program offers only one 
course in evaluation, and by definition is excluded from the directory. 
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Continuing programs: 

Delta University's School of Education offers degrees in policy, planning 
and evaluation through the Department of Administration and Policy Studies. 
The program originally started as Educational Research Methodology with a 
statistics emphasis but according to Dr. J, about 7 or 8 years ago we reorganized the 
department, and school, and we saw "evaluation as a need area." We had a 
"cluster of experience" in the faculty in evaluation. Thoy kept teaching more and 
more evaluation classes and it gradually evolved into to an emphasis on 
evaluation, rather that educational research or statistics. The program has 
between 12 and 15 full-time PhJD. students, with about 12 of them supported 
financially. One faculty member said that this program is not projected to be a 
growth area. "I see it as stable, i£ we continue to have the outside funding." He 
believes it will continue to have strong students and faculty. 

Mountain University really has two evaluation programs: one in the 
Psychology Department, and one in the College of Education. The programs 
started in the late 1970's. The Psychology Department offers a Ph.D. in 
research and evaluation methodology; the College of Education offers an 
Ed.D. in evaluation. The psychology program has about 13 students; the 
interdisciplinary program in the College of Education has at most two. It has 
recently hired a faculty member to fill an area the program had seen as a weak 
point. One faculty member noted that her students had an early opportunity 
to get out in the field. She felt this made for a strong program. The faculty 
have published extensively, including at least one textbook in evaluation. It 
has proven to be attractive to faculty, with several wanting to join; it is also 
popular with the administration. The faculty see it as a strong program with 
solid institutional support. 



21 



A3 



The 1994 AEA Annual Conference 



Private Eastern University's evaluation program began in the late 
1970's. In 1984, the faculty were given offices together and came to be 
recognized as a program area in the college by other faculty. Its strengths 
come from a diversity of faculty interests and beliefs, the enthusiasm of the 
faculty for evaluation teaching it, involving its graduate students in 
professional activities, and the fact that faculty are not dependent upon the 
funding of grants to maintain the program. One faculty member anticipates 
more involvement with international development and health services 
areas; another predicts a revamping of the curriculum to allow for 
modularized components of classes, more outreach to business and industry, 
and use of leading edge technology (like CD ROM-based simulations) to 
provided advanced training comparable to that now available in business 
settings. 

Pond State University is the major university of its state. The 
evaluation program had began in 1972 under the title of educational research 
as a single course and an occasional seminar. In 1982, the term 'evaluation' 
was added to the title, a move that, according to one faculty member, gained 
"face validity." It now offers six core classes on theory as well as statistics, 
research design, log-linear analysis, and modeling. An additional course in 
independent study or practicum is also required. One faculty member noted, 
"Publicity of the offerings is spreading, and we get students sent over from 
other departments. The other programs offer minors in evaluation" He said 
that they need to fight for the program regularly. "We have few resources 
and tight finances. We have about l/4th of the total graduate students in the 
College of Education, but we get nothing like that level of support." He also 



Appendix A 

explained that the university offers early retirement with benefits, and some 
of the faculty are looking into it. "If we pulled out, then what?" He can't see 
the university replacing the faculty en mass if these retirements occurred. 
The program would probably be gone if this early retirement scenario was 
carried out. 

Tree State has between 10 and 15 full-time Ph.D. students working full- 
time to obtain a degree with a concentration in educational evaluation, 
measurement, and research design. Although the program began in 1973, the 
Ph.D. portion has only existed for six years. Tree State has both a strong, well- 
known faculty and several centers that focus on educational evaluation. Two 
of the faculty have part time appointments in these centers. One faculty 
contact reported that all his outside appointment leaves him time for is 
advising doctoral students and sitting on committees. This program requires 
extensive research and outside experience. It is no wonder that "we expect 
students to take 3 1/2 to 4 years to complete the program." One faculty 
member stated that they have a fairly small program and don't actively 
recruit. They would like to expand their training and recruiting, which 
would increase the numbers of students. They would like to provide more 
stipends for the students and continue to provide the hands-on experience. 
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